
NOISE FILTERING AN IMAGE SEQUENCE 



BACKGROUND OF THE INVENTION 
Field Of The Invention 

[0001] The invention relates to noise filtering an image 
sequence. The invention further relates to encoding an image 
5 sequence, wherein the image sequence is noise filtered. 



Description Of The Related Art 

[0002] It is well known that image sequences generally contain 
noise that may arise either during the initial stage of image 

10 acquisition, or during the processing and transmission operations, 
or even during the storing stage. This noise not only degrades the 
quality of the sequence but also the performance of subsequent 
possible compression operations (e.g., MPEG, wavelet, fractal, 
etc.). For these reasons, there is great interest in reducing the 

15 noise as much as possible without unacceptably affecting the image 
quality. 

[0003] To reduce the noise, a filtering operation is necessary. 
Such a filtering operation may result in blurring and 'ghost' 
effects in the image, resulting in an unacceptable quality for the 
20 viewer. This is due to the fact that almost all images have 
detailed areas, with edges, contours, etc. 
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SUMMARY OF THE INVENTION 
[0004] An object of the invention is to provide advantageous 
filtering. To this end, the invention provides a method and device 
for noise filtering an image sequence, and a method and device for 
5 encoding an image sequence. 

[0005] In a first embodiment of the invention, statistics in at 
least one image of the image sequence are determined, and at least 
one filtered pixel value is calculated from a set of original pixel 
values obtained from the at least one image, wherein the original 

10 pixel values are weighted under control of the statistics. The 
invention provides a simple method to perform an adaptive 
filtering, which is preferably applied in a pre-processing stage of 
a compression system. Statistics may be easily obtained from the at 
least one image by any known (or yet unknown) calculation, e.g., 

15 variance or correlation (or approximation thereof) in a (sub-set) 
of the at least one image. 

[0006] In a further embodiment of the invention, the step of 
calculating comprises weighting the set of original pixel values 
under control of the statistics to obtain a weighted set of pixel 

20 values, and furnishing the weighted set of pixel values to a static 
filter for calculating the at least one filtered pixel value from 
the weighted set of pixel values. This embodiment has, inter alia, 
the advantage that adaptivity of the filtering is obtained by using 
a separate weighting step, and that a static filter is used in 

25 combination with the weighting. Instead of using a variable filter, 
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which implementation is more complicated, the invention provides a 
simple adaptation of the pixel values which, in combination with a 
static filter, results in adaptive filtering. 

[0007] Advantageously, the statistics include a spatial spread 
5 and/or temporal spread of the set of original pixel values. In this 
embodiment, the adaptation is based on the computation of a 
'spread' of the pixel values that are processed to obtain a 
filtered pixel value. The spread is a measure based on differences 
between pixel values, the spread being preferably computed as a sum 

10 of absolute differences, a given absolute difference being obtained 
by subtracting an average pixel value from a given original pixel 
value. The local 'spread', i.e., the spread of the set of original 
pixels from which a filtered pixel value is calculated, is a good 
indicator of the local activity of the image. In this way, on the 

15 basis of the statistics of the pixels that are processed, it is 

possible to control locally the strength of the filter in order to 
prevent annoying artifacts where the image content is critical, 
e.g., on the edges. In pre-f iltering, i.e., before entering a 
coding loop, defects around the moving objects and, in particular, 

20 the moving edges are eliminated by means of the adaptivity, based 
on the local statistical properties of the images in order to 
accomplish spatial filtering and also spatio-temporal filtering 
capable of being strongly effective against white Gaussian noise, 
without producing unacceptable artifacts in the image sequence. 
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This is especially true when averaging filters are applied. Median 
filtering reduces both Gaussian and spiky noise. 

[0008] Advantageously, the weighted pixel values are obtained by 
taking, for each pixel in the set of original pixels, a combination 
5 of a portion a of the original pixel value, and a portion 1-a of 
the central pixel value. In fact, a indicates the amount to which 
the original pixel values take the value of the central pixel 
value. In case, a = 0, all original pixel values have the same 
value as the central pixel value, i.e., the original pixel values 

10 other than the central pixel value are not taken into account. This 
is preferably the case when the local spread is high. In case a = 
1, all original pixel values keep their the original value. This is 
preferably the case when the local spread is low. In general, the 
higher the spread, the lower the value of a. In this embodiment, 

15 the control signal consists of only one value, i.e., a, so that the 
implementation can be kept as small as possible. 
[0009] The local spread is preferably furnished to a look-up 
table, whose output controls the weighting. A look-up table 
provides a simple and fast way of controlling the weighting. 

20 [0010] Preferred filtering operations in the present invention 
include median filtering and averaging filtering. When spread in 
temporal direction is used, e.g., in a spatio-temporal averaging 
filtering, it is preferred to use a second look-up table for the 
temporal direction, because pixel values in temporal directions are 

25 often differently correlated to each other than pixel values in 
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spatial directions. Further, pixels in the temporal directions are 
less correlated to pixels in the spatial directions. Therefore, it 
is advantageous to lessen the weight of neighboring pixels in the 
temporal directions in the total result in comparison to pixel 
5 values in the spatial directions, 

[0011] In case a temporal direction is used, the temporally 
displaced original pixel values preferably include two original 
pixel values from different fields (with unequal parity) in a same 
frame and at least one original pixel value of a previous frame. 
10 This embodiment saves memory compared to storing pixel values of 
fields with same parity in different frames, because in the latter 
case, at least two frames need to be stored to have two fields 
available . 

[0012] Further, filtered temporally displaced pixel values may 
15 be used rather than temporally displaced original pixel values to 
reduce bandwidth requirements of the implementation of the filter. 
[0013] U.S. Patent 5,621,468 discloses a motion adaptive spatio- 
temporal filtering method which is employed as a pre-filter in an 
image coding apparatus, which processes the temporal band- 
20 limitation of the video frame signals on the spatio-temporal domain 
along the trajectories of a moving component without temporal 
aliasing by using a filter having a band- limitation characteristic 
according to a desired temporal cutoff frequency and the velocity 
of moving components. 
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[0014] U.S. Patent 4,682,230 discloses an adaptive median filter 
system, which filters samples of an input signal. Further circuitry 
estimates the relative density of the noise in the input signal to 
generate the control signal supplied to the adaptive median filter. 
5 The adaptive filter selectively substitutes the sample having the 
median value for the current sample. If the current sample/median 
distance exceeds the processed inter M-tile distance, then the 
median valued sample is coupled to the output, and otherwise, the 
current sample is coupled to the output. M-tile is a generic term 

10 relating to the relative position of a sample in a list of samples 
sorted according to their value. The median and upper and lower 
quartiles are special cases indicating values one-half, three- 
quarters and one-quarter of the way through the ordered list, 
respectively. The inter M-tile distance is the difference between 

15 the upper M-tile value and the lower M-tile value and is a measure 
of the contrast of the image in the locality of the current sample. 
[00151 U.S. Patent 5,793,435 discloses de- interlacing of video 
using a variable coefficient spatio-temporal filter. The interlaced 
video signal is input to a video memory, which, in turn, provides a 

20 reference and plurality of offset video signals representing the 
pixel to be interpolated, and spatially and temporally neighboring 
pixels. A coefficient index, transmitted with the interlaced video 
as an auxiliary signal, or derived from motion vectors transmitted 
with the interlaced video, or derived directly from the interlaced 

25 video signal, is applied to a coefficient memory to select a set of 
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filter coefficients. The reference and offset signals are weighted 
together with the filter coefficients in the spatio-temporal 
interpolation filter, such as a FIR filter, to produce an 
interpolated video signal. The interpolated video signal is 
5 interleaved with the reference video signal, suitably delayed to 
compensate for filter processing time, to produce the progressive 
video signal. 

[0016] The aforementioned and other aspects of the invention 
will be apparent from and elucidated with reference to the 
10 embodiments described hereinafter. 

BRIEF DESCRIPTION OF THE DRAWINGS 
[0017] In the drawings: 

[0018] Fig. 1 shows an embodiment of an encoder according to the 
15 invention; 

[0019] Fig. 2 shows input samples of adaptive filters as shown 
in Figs . 3 and 4 ; 

[0020] Fig. 3 shows an embodiment of an adaptive spatial median 
filter according to the invention; 
20 [0021] Fig. 4 shows an embodiment of an adaptive spatial 
averaging filter according to the invention; 

[0022] Fig. 5 shows a first set of input samples of an adaptive 
spatio-temporal averaging filter as shown in Fig. 6; 
[0023] Fig. 6 shows an embodiment of a spatio-temporal averaging 
25 filter according to the invention; and 
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[0024] Fig. 7 shows a second set of input samples of an adaptive 
spatio-temporal averaging filter as shown in Fig. 6. 

[0025] The drawings only show those elements that are necessary 
to understand the invention. 

5 

DESCRIPTION OF THE PREFERRED EMBODIMENTS 
[0026] Fig. 1 shows an embodiment of an encoder 1 according to 
the invention, comprising an input unit 10, a computing unit 11, a 
look-up table 12, a weighting stage 13, a filter 14 and an encoding 

10 unit 15. An input video signal VI is furnished to the encoder 1 and 
received in the input unit 10. In computing unit 11, a local spread 
S is obtained from a set of original pixel values indicated by P^, 
M±. The result of the spread computation is furnished to the look- 
up table 12 to obtain a control signal a. In the weighting stage 

15 13, the pixel values Pt, Mj^ are weighted to obtain weighted pixel 
values Pt, N±. The weighted pixel values P^, N± are filtered in the 
filter 14 to obtain a filtered pixel value P^'. A plurality of 
pixel values P^' constitute a filtered video signal. According to 
advantageous embodiments of the invention, the filter 14 includes a 

20 spatial median filter, a spatial averaging filter, a spatio- 
temporal averaging filter, or a combination of these. The filtered 
video signal, constituted of the plurality of filtered pixel values 
Pt', is encoded in the encoding unit 15 to obtain an encoded video 
signal V2. The encoding unit 15 is preferably an MPEG encoder. 
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[0027] Fig, 2 shows exemplary input samples of an adaptive 
filter according to the invention, e.g., a spatial median filter as 
shown in Fig. 3 or a spatial averaging filter as shown in Fig. 4. 
These input samples may also be used in a preferred example of 
5 input samples within one field. Dotted lines indicate image lines 
of a first field and continuous lines indicate image lines of a 
second field of a frame. A sample is at a position of a 
calculated output sample. To calculate one filtered luminance 
sample, five samples P^/ M3 and M4 are used as input. In an 

10 MPEG encoder, which is a preferable field of application of the 

invention, horizontal color sub-sampling has normally already taken 
place at the input, according to the CCIR 4:2:2 format. Therefore, 
a horizontal distance between color samples (Ptc/ ^2c' %c 

for U&V) is twice as large as for the luminance samples. 

15 Because experiments indicated that extra gain from the color 

samples is minor, color median processing can be skipped without 
significantly losing quality. 

[0028] Median filtering per se is known in the art for its 
capability of preserving monotonic step edges, and is therefore 

20 widely used for two-dimensional image noise smoothing. The 

implementation of a median filter requires a very simple digital 
non-linear operation: a sampled and quantized signal of length n is 
taken; a window that spans m signal Scunple points is slid across 
the signal. The filter output is set equal to the median value of 

25 these m signal samples and is associated with the sample at a 
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center of the window. The median of m scalar X± with i = 1, ...,m 
can be defined as the value X^^ed such that, for all Yi 

m m 

1L\X„^-X\<Y,\Y-X\ (1) 

1=1 1=1 

[0029] In order to obtain a unique value as a result, m must be 
an odd value. Suppose a random sample [X^, . . . ,Xj^} from a population 
having a bi- exponential density function described by the 
expression: 

(2) 

where y is a scaling factor and 5 is a maximum location parameter. 
The value of 5 maximizing the likelihood function: 

1=1 ^ 

is called the maximum likelihood estimate for 5, based on the 
random sample {X^, . . . ,X;n} . By taking a logarithm of (3), it can be 

20 shown that the maximum likelihood estimate is clearly equal to 
MediXi, . . . ,X^] . The median is thus an optimal estimate of the 
location parameter in the maximum likelihood sense, if the input 
distribution is double exponential as in (2) . In a similar manner, 
an average is the maximum likelihood estimate for a Gaussian 

25 distribution. 
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[0030] Conventionally, when the median filter is used for two- 
dimensional images, the intensity at every point in the image is 
replaced by the median of the intensity of the points contained in 
an m*m window centered at that point. It is known that the median 
5 filter is more effective than a linear filter for smoothing images 
with spiky noise distribution, because outliers are rejected by the 
median filtering. According to the properties mentioned above, the 
median filter tends to produce lower variances for the filtered 
noise when the distribution of the input noise has larger tails 

10 (e.g., spiky noise), but has lower performances then, e.g., an 

averaging filter in case of uncorrelated (white) image noise with a 
Gaussian distribution; also when either Gaussian or impulsive noise 
are present, the latter is not completely suppressed as when only 
impulsive noise is present. 

15 [0031] It has already been said that median filters are 

attractive for their capability of preserving monotonic step edges 
(width (in+l)/2) in the images, while an averaging filter tends 
unavoidably to blur edges, but is more effective against Gaussian 
noise. In an embodiment of the invention, a simple and easy 

20 implementation in real hardware is obtained by using a separable 
median filter. Such a separable filter performs median filtering 
operations by means of successive applications of one -dimensional 
median filters along different directions. Although the result is 
not identical to the full two dimensional median filter (using an 

25 /n*/n window) , it can be seen that the separable filter provides 



PHNL000345-SS- 100605 



11 



PHNL 000345 

comparable performances to the two-dimensional median filter. 
However, the main advantage is that in the full two-dimensional 
median filter, the center element is the median of points; by 
performing the median of m points separately along rows and 
5 columns, a computational saving factor can be achieved. Separable 
median filters as such are known in the art. 

[0032] Although the median has a good capability of preserving 
edges, if it is applied directly on the image data, strange effects 
can occur as blurring and 'tails' or * shadows' around moving parts. 

10 Inter alia, in order to minimize these undesired effects, the 

present invention provides an adaptive median filter, this filter 
being adaptive on the basis of local statistics of the image. 
[0033] Fig. 3 shows an embodiment of an adaptive median filter 
according to the invention. The input samples P^, M^^ as shown in 

15 Fig. 2, are furnished to a computing unit 21 and to a weighting 
stage 23. In the computing unit 21, a spatial spread Sgpat is 
calculated from the input samples, this spread Sgp^t being 
furnished to a look-up table 22. Based on the spread Sgp^t^ ^ 
control signal a is obtained from the look-up table 22. The control 

20 signal a is furnished to the weighting stage 23, in which the input 
pixel values P^, M± are weighted to obtain adapted pixel values P^/ 
N^. Note that in this embodiment, the central pixel P^ is 
unaffected by the weighting. In median filter 24, a median is taken 
from the adapted pixel values P^, N± to obtain a filtered pixel 

25 value Pt'- The median filter 24 comprises three separate median 
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filters 240, 241 and 242. These separate median filters 240, 241, 
242 together form a total median filter. The operation of this 
embodiment is discussed below. 

[0034] A spatial spread Sgpat of the five input samples P^/ ^i^ 
5 M2, M3 and M4 is computed as follows: 

5 



10 



20 



S ,= '-^ (5) 



The output of the spread of the luminance is translated, via the 
look-up table 22, into the control parameter a for the weighting 
stage 23, In a preferred embodiment, the content of the look-up 
table 22 is downloadable from an external source. An exemplary 
15 look-up table 22 is given by: 

Sspat >15=>a = 0.35 (6) 
S.pat>20^a = 0.2 



Adapted pixel values are then obtained by: 
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N,=cM,+(l-a)P, 
N,^aM^^{\-d)P, 

From these adapted pixel values, the median is computed in the 
5 filter 24 according to: 

P;^Med[Med{N,,N^,P,),P,,Med{N,,N,,P,)\ (8) 

As will be easily understood by those skilled in the art, the 
10 median is alternatively obtained by: 

P,^^Med[N,,N,,P,,N,,N,)\ (10) 

[0035] An advantage of a median filter according to the 
15 invention, e.g., the median filter 24 as discussed above, is that a 
gradual filtering is obtained around the edges so that annoying 
effects in the sequence are avoided, or, at least, attenuated. When 
the spread Sgpat larger, i.e., high spatial activity, e.g., 
around edges, then a is smaller so that the original central pixel 
2 0 is assigned a higher weight and the filtering of the median filter 
24 is weaker. 

[0036] Fig. 4 shows an embodiment of an adaptive spatial 
averaging filter according to the invention. Computing unit 31 and 
look-up table 32 are similar to computing unit 21 and look-up table 
25 22 as shown in Fig. 3. The look-up table 32 is coupled to a 
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weighting stage 33, in which the input samples Pt, M± are weighted 
to obtain adapted pixel values P^, N± that are furnished to a 
spatial averaging filter 34. 

[0037] As stated before, a spatial averaging filter is the 
5 maximum likelihood estimate for the Gaussian distribution. Since 
noise present in video sequences is usually a sum of effects due to 
different sources (acquisition, pre-amplifying, amplifying, 
transmission and handling operations) , it can be assumed in a lot 
of cases that the noise distribution is Gaussian (theorem of the 

10 central limit) , In these cases, an averaging filter is preferred. 

By using an adaptive averaging filter according to the invention in 
a pre- filtering stage of an encoding arrangement, effective noise 
filtering is obtained which results in a significant bit-rate 
reduction. However, it is necessary to pay attention to the quality 

15 of the resulting image, since blurring of the spatial and temporal 
edges unavoidably occur. An object of the invention, in relation to 
averaging filters, is to control such blurring in order to achieve 
an acceptable quality for the filtered sequence. For an adaptive 
spatial averaging filter, the adaptivity based on local statistical 

20 properties (spread/activity) of the image can be exploited as it 

has been described for the median filter. The result is an adaptive 
spatial averaging filter, which better preserves the quality of the 
images . 

[0038] Computation of the adapted pixel values is similar to the 
25 computation previously described in relation to the adaptive median 
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filter. Also, in this case, the filtering of the chrominance may be 
skipped, because its contribution to the final result is minor. 
[0039] An output of the adaptive spatial averaging filter is 
computed as follows: 

5 

^,, {N,^N,^N,I2^NJ2^P,) ^^^^ 
4 

It is noted, that the pixels and N4 are divided by a factor 2 to 
reduce their weight in the final average because their distance to 

10 Pt is double compared to Ni and N2, since the filtering is applied 
within a field, and are therefore 'less correlated' . 
[0040] When a very low level of noise is present, the image 
loolcs obviously much smoother than the original; anyway, by means 
of a proper adjustment of the look-up table, this effect can be 

15 statically controlled, achieving a good trade-off between the noise 
reduction and the good quality of the video sequence. 
[0041] Fig. 5 shows input samples in both spatial and temporal 
directions in which t denotes time. In frame Fq^ a set of pixels 
^t/ is taken, similar to the luminance pixels in Fig. 2. In 

20 addition, in this embodiment, pixel values P^i and Pt2 are taken 

from fields with the same parity in both a previous frame F_2 and a 
future frame F^. Here, a window of seven pixels is considered: five 
pixels of the present field, one pixel of the previous field with 
same parity and one of the future field with same parity. It is 

25 advantageous to include filtering operations in the temporal 
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direction, because both spatial and temporal noise are often 
present. A reduction of the level of noise can be useful for motion 
estimation either, provided that the motion estimation itself is 
thought and realized strictly related with the pre-processing part 
5 and consequently no too much affected by the increased smoothness 
of the filtered image, otherwise the quality of the motion vectors 
can be worse, resulting in some additional coding noise that 
compromises the final result. 

[0042] Fig. 6 shows an embodiment of a spatio-temporal averaging 

10 filter according to the invention. In order to reduce annoying 

effects, such as 'tails', 'shadows' or simply blurring in moving 
objects, an adaptation step is used in order to perform an 
effective and not image -damaging averaging spatio-temporal 
filtering. Also, in this case, the adapt ivity is based on the local 

15 statistical properties of the image, even if it is now necessary to 
make a distinction between the pixels belonging to the same field 
and the pixels belonging to the previous or the next field with 
same parity. The embodiment comprises a computing unit 41 for 
computing a spatial spread which is similar to computing units 21 

2 0 and 31 as shown in Figs. 3 and 4. The computing unit 41 is coupled 
to a look-up table 43. In this exemplary embodiment, a spread of 
the pixels belonging to the same field (Pt, Mj^) and a spread of the 
pixels (Pf -Pfcl' ^t2) belonging to different fields with same parity 
are separately computed. In other words, the computation of the 

25 spread in spatial directions is separated from the computation of 



PHNL000345-SS- 100605 



PHNL 000345 



the spread in the temporal directions. For computing the temporal 
spread Stemp the embodiment comprises a second computing unit 42. 
[0043] The temporal spread is computed as follows: 



3 

2 



S.™p= ^ (13) 



The result of the temporal spread is translated via a temporal 
10 look-up table 44 into a control parameter a' necessary to perform 
weighting operations on the temporal pixel values P^, Ptl ^12- 
[0044] After the computation of the control parameters a 
(spatial) and a' (temporal) , the weighting operation is performed 
in both the spatial and temporal direction, in the spatial 
15 direction according to the formula (7) and in the temporal 
direction according to: 

WP,=a^P,,+(\-a')P, ^^^^ 
WP^ =a'P,^ +{\-a')P, 

20 Finally, an output of the spatio-temporal averaging filter 47 is 
computed according to: 

p,„_ iN,+N,+N,/2 + NJ2 + P,+WPJa + WP,/a) ^^^^ 
' 4 + 2/a 
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[0045] Note that the weighted pixel values WPi and WP2 are 
divided by a control parameter a. The control parameter a is 
obtained from a look-up table 45 and is a number > 1, depending on 
the local temporal spread in the three pixels P^/ Ptl ^t2- the 

5 higher the spread, the higher a, so that the weight of the previous 
and the next pixel in the average is smaller. By adjusting the 
look-up table 45 properly, it is possible to control the strength 
of the filter in the temporal direction in order to achieve a good 
quality of the image, once again exploiting the adapt ively to the 
10 image temporal content so that annoying effects connected with 
edges blurring are reduced. 

[0046] The described filter belongs to the class of Finite 
Impulse Response (FIR) filters. The FIR structure requires keeping 
in memory the present Fq, the future Fj and the previous F_i 

15 original frames for the filtering operation. In order to save 
memory, it is preferred to use pixels of past fields and with 
unequal parity, as shown in Fig. 7. In this case, only the present 
Fq and the previous frame F_2 have to be stored. This allows a 
reduction of the memory size as far as the implementation of the 

20 filter is concerned, without significantly affecting the resulting 
quality of the filtered image. Instead of previous original frames, 
previous filtered frames may be used. In the case that for previous 
frames in Fig. 7, filtered frames are taken, an Infinite Impulse 
Response (IIR) filter structure is obtained. This structure has 

25 advantages regarding memory usage and bandwidth. 
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[0047] Examples of devices that encode an image sequence, in 
which noise filtering according to the invention is applied, are: 
MPEG-2 encoders, digital video recorders (e.g., DVD-video 
recording, digital -VHS, HDD VCR) , etc. 
5 [0048] Adaptive filters according to this invention may also be 
applied inside a motion- compensating coding loop. Advantageously, 
an adaptive filter is used in a pre- filtering stage in combination 
with a temporal filter within the coding loop. 

[0049] In an embodiment of the invention, at least two adaptive 
10 noise filters are combined, e.g., a spatial median filter and an 
adaptive spatial averaging filter, wherein the filtering is 
controlled by characteristics of the image sequence. A noise 
estimator may be added that analyses the level of the present 
noise. Such a noise estimator is an interesting tool to control the 
15 adaptive filters. Advantageously, the noise estimator is arranged 
to identify the statistical properties of the present noise in 
order to opportunely switch dynamically between the median and the 
spatial and/or spatio-temporal averaging filter. 

[0050] It should be noted that the above-mentioned embodiments 
20 illustrate rather than limit the invention, and that those skilled 
in the art will be able to design many alternative embodiments 
without departing from the scope of the appended claims. In the 
claims, any reference signs placed between parentheses shall not be 
construed as limiting the claim. The word 'comprising' does not 
25 exclude the presence of other elements or steps than those listed 
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in a claim. The invention can be implemented by means of hardware 
comprising several distinct elements, and by means of a suitably 
programmed computer. In a device claim enumerating several means, 
several of these means can be embodied by one and the same item of 
5 hardware. The mere fact that certain measures are recited in 
mutually different dependent claims does not indicate that a 
combination of these measures cannot be used to advantage. 
[0051] In summary, noise filtering an image sequence is provided 
wherein statistics in at least one image of the image sequence is 
10 determined and at least one filtered pixel value is calculated from 
a set of original pixel values obtained from the at least one 
image, wherein the original pixel values are weighted under control 
of the statistics. 
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